Evaluating the Quality of Multilingual Items Generated Using Automatic Processes: Preliminary Results from a Reliability Study

Introduction The traditional approach to test item translation is an effortful, time-consuming process conducted by bilingual or multilingual content specialists. One important problem that arises when content specialists perform translations is the introduction of subjectivity into the process. For instance, Hambleton (1993) reported that when a content specialist is told that another specialist will be back translating his or her work, the content specialist selects words that are more likely to be translated into the original item. Of course, translations can also be evaluated using other methods such as administering the original and the translated test items to a group of specialists who are fluent in both languages. However other issues may arise such as the possibility that the translators are stronger in one language compared to another (Hambleton, Merenda, & Spielberger, 2005). As the implementation of technology in test development grows, the emergence of automatic item generation (AIG; Gierl & Haladyna, 2013) with the use of item models (Gierl, Alves, & Zhou, 2008) is becoming more prominent. The translation of item models may make it possible for large numbers of items in two or more languages to be generated simultaneously. To-date, the quality of such translation has yet to be evaluated. Hence, the purpose of our study is to begin this evaluative process by determining the similarity of the translated items when generative models are used to create items in multiple languages. We will also attempt to take advantage of the available technology for translation by using the online engine Google Translate. With the growing demand for computer-based testing and computer adaptive testing, a large number of items are now required to permit flexible test administration while maintaining adequate item security. When more items are needed for this item banking model, the traditional approach of item development becomes effortful and time consuming and it requires extensive financial resources. Traditionally, items are hand crafted by content specialists one-by-one to ensure certain content areas with appropriate skill levels are covered. But this process becomes expensive as more items are needed. The use of Automatic Item Generation (AIG) saves time and money by combining content expertise with computer technology to automatically create new items. The development of multiple-choice items in the medical context using AIG was recently described by Gierl, Lai, and Turner (2012) using a three-step process. In the first step, content specialists create a graphical representation known as a cognitive …